CONTRASTIVE LANGUAGE-IMAGE PRE-TRAINING

Papers with Contrastive Language-Image Pre-training

Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval (2025.emnlp-main)

Copied to clipboard

Tianlu Zheng, Yifan Zhang, Xiang An, Ziyong Feng, Kaicheng Yang, Qichuan Ding

Challenge:	a large-scale visionlanguage pre-training framework is limited by the scarcity of large-sized annotated vision-language data . noise-resistant data construction pipeline is needed to filter and caption web-sourced images . noisy text tokens can be a problem for fine-grained representation learning .
Approach:	They develop a noise-resistant data construction pipeline that leverages in-context learning capabilities of MLLMs to automatically filter and caption web-sourced images.
Outcome:	The proposed framework improves cross-modal alignment by masking noisy textual tokens based on the gradient-attention similarity score.

UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding (2023.findings-acl)

Copied to clipboard

Rui Sun, Zhecan Wang, Haoxuan You, Noel Codella, Kai-Wei Chang, Shih-Fu Chang

Language Models

Challenge:	supervised methods for vision-language tasks have been well-studied, but they lack the fine-grained information needed for semantics understanding.
Approach:	They propose a framework to take advantage of fine-grained information for zero-shot vision-language learning, covering multiple tasks such as VQA, SNLI-VE, and VCR.
Outcome:	The proposed framework outperforms previous zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR.

CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling (2025.emnlp-main)

Copied to clipboard

Jihai Zhang, Xiaoye Qu, Tong Zhu, Yu Cheng

Multimodality

Challenge:	Recent studies found that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features.
Approach:	They propose a model-agnostic approach that fine-tunes complementary CLIP models and transforms them into a CLIP-MoE.
Outcome:	The proposed framework fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE.

RWKV-CLIP: A Robust Vision-Language Representation Learner (2024.emnlp-main)

Copied to clipboard

Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, Jiankang Deng

Semantic Text Processing

Challenge:	Using large image-text datasets, large-scale image-data sets have been used for visionlanguage pre-training.
Approach:	They propose a framework that leverages Large Language Models to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags.
Outcome:	The proposed framework can combine and refine information from web-based image-text pairs, synthetic captions, and detection tags.

RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning (2022.emnlp-main)

Copied to clipboard

Yi Zhu, Zhaoqing Zhu, Bingqian Lin, Xiaodan Liang, Feng Zhao, Jianzhuang Liu

Semantic Text Processing

Challenge:	Existing visual relationship detection models only use numeric ids of relation labels for training, but ignore semantic correlation between labels.
Approach:	They propose a visual Relationship prediction framework that transfers natural language knowledge from Contrastive Language-Image Pre-training models to enhance the relationship prediction.
Outcome:	The proposed framework improves visual relationship prediction by matching semantic correlations with relation triplets.

Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data (2025.naacl-long)

Copied to clipboard

Haonan Wang, Minbin Huang, Runhui Huang, Lanqing Hong, Hang Xu, Tianyang Hu, Xiaodan Liang, Zhenguo Li, Hong Cheng, Kenji Kawaguchi

Language Models

Challenge:	Contrastive Language-Image Pre-training (CLIP) is a standard for cross-modal image-text representation learning.
Approach:	They propose a framework that enhances pre-trained CLIP models by exploiting challenging text-image pairs within existing datasets.
Outcome:	The proposed framework improves CLIP models by exploiting text-image pairs in training.

Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment (2025.emnlp-main)

Copied to clipboard

Tong Zhang, Kuofeng Gao, Jiawang Bai, Leo Yu Zhang, Xin Yin, Zonghui Wang, Shouling Ji, Wenzhi Chen

Language Models

Challenge:	Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are vulnerable to data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet.
Approach:	They propose an Optimal Transport-based framework to reconstruct image-caption pairs and propose an optimal transport-based distance measure to re-assign new captions based on the proposed optimal transport distance.
Outcome:	The proposed framework reduces the attack success rates of poisoning attacks to 0% in most cases.

Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision? (2025.acl-long)

Copied to clipboard

Zihao Li, Lecheng Zheng, Bowen Jin, Dongqi Fu, Baoyu Jing, Yikun Ban, Jingrui He, Jiawei Han

Multimodality

Challenge:	Graph Neural Networks (GNNs) with CLIP pipeline are difficult because of the scarcity of labeled data and text supervision, different levels of downstream tasks, and conceptual gaps between domains.
Approach:	They propose a multi-modal prompt learning paradigm to adapt pre-trained GNNs to downstream tasks with weak text supervision.
Outcome:	The proposed model can generalize graphs to unseen classes with weak text supervision.

Delving into the Openness of CLIP (2023.findings-acl)

Copied to clipboard

Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, Xu Sun

Visual Data in NLP

Challenge:	Contrastive Language-Image Pre-training (CLIP) allows for open-vocabulary visual recognition, where the model can recognize images from an open class set in a zero-shot manner.
Approach:	They propose to use image classification as an image-to-text matching task instead of discrete category IDs to achieve open-vocabulary visual recognition.
Outcome:	The proposed model can recognize images from an open vocabulary in a zero-shot manner, but its performance deteriorates as the vocabulary expands.

IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning (2023.findings-emnlp)

Copied to clipboard

Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, Babak Damavandi

Semantic Text Processing

Challenge:	Existing methods to align motion sensors with text and video are limited in their scale and limited in the use of IMU models.
Approach:	They propose to project IMU motion sensor recordings into the joint representation space of Contrastive Language-Image Pre-training (CLIP) they introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data.
Outcome:	The proposed approach significantly improves downstream performance when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource.

VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models (2024.emnlp-main)

Copied to clipboard

Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, Lianwen Jin

Multimodality

Challenge:	Existing studies have shown that CLIP models with short summary texts cannot process extensive textual descriptions due to its text encoder's reliance on positional embeddings with length 77.
Approach:	They propose a Contrastive Language-Image Pre-training (CLIP) model which aims to unleash the long-description understanding capability of video CLIP models.
Outcome:	The proposed model can learn the distribution of feature space while expanding the long description capability.